DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement
نویسندگان
چکیده
In this paper, we introduce a novel Java implementation of multiple inter-rater agreement measures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, π, κ, α, etc., we particularly support unitizing tasks by measuring αU as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.
منابع مشابه
Test-Retest and Inter-Rater Reliability Study of the Schedule for Oral-Motor Assessment in Persian Children
Objectives: Reliable and valid clinical tools to screen, diagnose, and describe eating functions and dysphagia in children are highly warranted. Today most specialists are aware of the role of assessment scales in the treatment of affected individuals. However, the problem is that the clinical tools used might be nonstandard, and worldwide, there is no integrated assessment performed to assess ...
متن کاملReliability of Body Landmarks Analyzer for Measuring the Quadriceps Angle
Genovarum and Genovalgum are the most common postural deformities of the knee joint. A quadriceps angle is used to measure these anomalies. Methods of measuring this angle are divided into two categories: invasive and non-invasive. The purpose of the present research was to study the inter/intra rater reliability of the non-invasive Body Landmarks Analyzer method for measuring of the quadriceps...
متن کاملA method for quantitative measurement of lumbar intervertebral disc structures: an intra- and inter-rater agreement and reliability study
BACKGROUND There is a shortage of agreement studies relevant for measuring changes over time in lumbar intervertebral disc structures. The objectives of this study were: 1) to develop a method for measurement of intervertebral disc height, anterior and posterior disc material and dural sac diameter using MRI, 2) to evaluate intra- and inter-rater agreement and reliability for the measurements i...
متن کاملImplications of Inter-Rater Agreement on a Student Information Retrieval Evaluation
This paper is about an information retrieval evaluation on three different retrieval-supporting services. All three services were designed to compensate typical problems that arise in meta-data-driven Digital Libraries, which are not adequately handled by a simple tf-idf based retrieval. The services are: (1) a co-word analysis based query expansion mechanism and re-ranking via (2) Bradfordizin...
متن کاملUltrasound assessment for grading structural tendon changes in supraspinatus tendinopathy: an inter-rater reliability study
AIM To evaluate the inter-rater reliability of measuring structural changes in the tendon of patients, clinically diagnosed with supraspinatus tendinopathy (cases) and healthy participants (controls), on ultrasound (US) images captured by standardised procedures. METHODS A total of 40 participants (24 patients) were included for assessing inter-rater reliability of measurements of fibrillar d...
متن کامل